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PREFACE 


The  work  described  in  this  report  was  authorized  under  the  interagency  agreement 
between  the  Environmental  Protection  Agency  and  U.S.  Army  Edgewood  Chemical  Biological 
Center.  This  work  was  started  in  March  2001  and  conq)leted  in  December  2001. 

The  use  of  eiOier  trade  or  manu&cturers*  names  in  this  report  does  not  constitute  an 
official  endorsement  of  any  commercial  products.  This  report  may  not  be  cited  for  purposes  of 
advertisement. 

This  report  has  been  approved  for  public  release.  Registered  users  should  request 
aHditinnal  copies  from  the  Defense  Technical  hiformation  Center;  unregistered  users  should 
direct  such  requests  to  the  National  Technical  Infomiation  Service. 
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BIO-DETECTOR  ASSESSMENT 


This  report  describes  the  Bio-Detector  assessment  that  the  Edgewood  Chemical 
Biological  Center  performed  for  the  Environmental  Response  Team.  The  document  is  composed 
of  seven  parts: 

1)  Background  -  a  brief  summary  of  the  objective  and  requirements  of  the  study. 

2)  Process  Description  -  a  description  of  the  approach  and  evaluation  methodology  that 
was  followed  to  conduct  the  study. 

3)  Bio-Detector  Devices  -  a  listing  of  the  devices  that  were  considered  for  assessment. 

4)  Evaluation  Model  -  a  description  of  the  criteria  that  were  used  to  evaluate  the 
devices. 

5)  Evaluation  Results  -  information  on  how  the  devices  rank  based  on  the  evaluation, 
including  a  discussion  of  technology  strengths  and  weaknesses. 

6)  Conclusions  and  Recommendations  —  a  proposed  approach  to  meet  the  stated 
objectives,  based  on  analysis  of  the  results. 

7)  Appendices  -  various  information  such  as  a  listing  of  study  participants  and  more 
detailed  analysis  of  the  results. 


1.  BACKGROUND 

The  Federal  Response  Plan  charges  the  Environmental  Protection  Agency  (EPA)  with  the 
mission  to  respond  to  spills  or  releases  of  hazardous  materials  and  handle  health,  safety,  and 
clean-up  actions.  EPA  Environmental  Response  Teams  (ERT)  would  serve  as  technical  advisors 
to  On-Scene  Coordinators  for  these  incidents.  The  ERT  identified  two  scenarios. 
Decontamination  and  Identification,  to  describe  their  responsibilities.  Decontamination  has  the 
highest  priority  and  has  the  objectives  of  determining  effectiveness  of  the  decontamination 
operation  as  well  as  the  impact  to  the  environment  from  decontamination  operations.  The 
identification  scenario  is  defined  as  a  confirmation  of  prior  analyses. 

The  ERT  would  like  to  improve  their  capability  to  respond  to  incidents  involving  agents 
of  biological  origin.  To  accomplish  this,  they  would  need  the  ability  to  handle  samples  in  all 
forms  (air,  liquid,  solid,  vegetation)  and  to  quickly  perform  presumptive  analysis  with  an  on-site, 
portable  system.  The  ERT  tasked  the  US  Army  Edgewood  Chemical  Biological  Center  (ECBC) 
to  provide  guidance  on  how  the  ERT  could  implement  a  biological  detection  capability  to 
support  their  decontamination  mission  under  the  Federal  Response  Plan  and  to  also  provide  a 
limited  analysis  capability  to  support  a  wider  mission. 

The  objective  of  the  ERT  Bio-Detector  assessment  was  to  evaluate  available  biological 
detection  and  identification  devices  for  their  potential  to  meet  EPA  program  needs,  and  to 
recommend  the  best  candidate(s)  for  further  testing.  The  study  was  structured  to  address  the  two 
scenarios  described  above,  resulting  in  separate  assessments  being  performed  for  Detector  and 
Identifier  devices. 
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2. 


PROCESS  DESCRIPTION 


A  structured  decision  analysis  process  was  developed  for  this  study.  The  q)proach  has 
been  used  for  numerous  similar  efforts  at  ECBC.  At  its  core  is  the  identification  of  important 
^ors,  or  mission  essential  criteria,  against  which  candidate  devices  are  evaluated.  The  process 
includes  thorough  documentation  of  the  results  and  the  associated  rationale  so  that  final  results 
and  recommendations  can  be  readily  explained  and  defended. 

The  process  is  summarized  below: 

1  •  Formation  of  study  team  —  The  study  team  consisted  of  three  groups  of  participants: 

•  Customer  (ERT)  representatives  -  These  participants  were  responsible  for 
articulating  customer  objectives,  needs,  and  requirements. 

•  Technical  subject  matter  experts  (SME’s)  -  These  participants,  referred  to  as  the 
evaluation  panel,  have  detailed  knowledge  of  the  devices  (and  the  technologies  iqmn 
Anfoich  they’re  based)  that  were  evaluated.  They  also  support  similar  missions  within 
ECBC  and  have  an  understanding  of  operational  necessities. 

•  Decision  Analysis  Team  (DAT)  -  The  DAT  guided  the  study  team  through  the 
process.  This  included  the  conduct  of  evaluation  meetings  and  the  evaluation  and 
documentation  of  results. 

The  study  team  partic^ated  in  evaluation  meetings  that  were  fecilitated  by  the  Decision 
Analysis  Team.  A  laptop  con:5)uter  and  projector  were  used  to  focus  the  participants’ 
deUberations  and  to  document  information  during  the  evaluation  meetings.  All  decisions 
were  made  by  consensus.  The  members  of  the  study  team  are  listed  in  Appendix  A. 

Identify  and  describe  candidate  devices  -  The  devices  that  had  potential  to  meet  customer 
needs  were  identified.  Detailed  descr^tions  of  the  devices,  to  include  technical  capabilities 
and  level  of  maturity,  physical  characteristics,  and  operational  considerations  were  also 
obtained  and  provided  to  the  study  team.  The  devices  that  were  considered  are  listed  in 
Section  3  of  the  report. 

3.  Develop  evaluation  model  -  An  evaluation  model  was  developed  to  evaluate  the  devices. 

The  core  of  the  model  was  the  evaluation  criteria  that  were  derived  fi’om  customer 
needs/requirements.  Definitions  and  performance  scales  were  developed  for  each  criterioa 
The  model  and  how  it  was  developed  are  described  in  Section  4  of  the  report. 

Evaluate  the  candidate  devices  -  The  candidate  devices  were  scored  against  all  criteria,  using 
the  performance  scales.  The  technical  subject  matter  experts  first  individually  scored  the 
candidates,  and  then  convened  as  an  evaluation  panel  to  do  consensus-based  scoring.  The 
assessments  were  based  on  available  data,  as  well  as  the  experience  and  expertise  of  the 
panel  members.  A  decision  support  software  tool.  Logical  Decisions  for  Windows  (LDW), 
was  used  to  develop  and  document  the  evaluation  model,  elicit  and  capture  weights  and 
scores,  and  to  aid  in  the  analysis  of  results. 
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5.  Analwe  resufts  -  LDW  generated  rankings  that  are  based  on  the  technology  scores  and  the 
criteria  weights.  Each  candidate  device  was  analyzed  for  individual  strengths  and 
weaknesses,  and  the  results  were  summarized.  Soisitivity  analysis  was  also  performed  to 
determine  >\diich  factors  had  the  greatest  impact  on  the  results.  The  results  and  related 
analysis  is  contained  in  Section  5  and  Appendix  C  of  the  report. 

6.  Document  results  and  recommendations  -  The  analysis  of  results  led  to  the  recommendations 
that  are  provided  in  Section  6  of  the  report. 


3.  BIOLOGICAL  DETECTION  DEVICES 

A  literature  search  was  conducted  to  identify  devices  that  had  the  potential  to  meet  EPA 
needs.  Descr4)tions  of  each  device  were  prepared  and  provided  to  the  study  team  (those 
descriptions  are  available  as  a  separate  document).  The  devices  were  grouped  according  to 
whether  they  were  detectors  or  identifiers. 

In  this  context,  detectors  are  devices  that  can  be  used  to  determine  if  a  material  is  of 
biological  origin.  To  acconqjlish  this,  the  device  may  detect  the  presence  of  DNA,  protein,  or 
adenosine  tri-phosphate,  which  is  the  energy  source  in  the  ceU.  The  presence  of  any  one  or 
several  of  these  markers  suggests  that  the  material  could  be  of  biological  origin.  Additionally, 
these  detection  devices  can  be  used  to  determine  if  the  biological  material  in  the  sample  is  live 
(viable).  This  viability  issue  is  important  for  decontamination  issues. 

Within  the  identification  category,  the  devices  were  described  as  being  immunoassay- 
based  or  nucleic  acid-based  technologies. 

Immimoassay  technology  is  based  on  the  use  of  antibodies  that  target  proteins  unique  to  a 
biological  agent.  It  has  been  used  as  a  field  anal3^is  technique  in  both  clinical  and  environmental 
roles  for  many  years.  Rapid  immunoassays  have  been  used  since  the  early  1980’s  to  directly 
detect  microbiological  agents  in  clinical  specimens.  The  1990’s  saw  application  of  this 
technology  to  environmental  uses,  including  pesticide  analysis. 

Nucleic  acid-based  technologies  accomplish  detection  and  identification  through  the 
targeting  of  nucleic  acids,  usually  DNA,  of  biological  agents.  It  was  developed  during  the  mid- 
1980’s  as  a  clinical  analysis  teclmique  and  its  uses  are  still  expanding.  The  technology  offers  the 
most  specific  and  sensitive  analysis  for  microbiological  materials  in  environmental  samples; 
however,  it  is  also  the  most  prone  to  interferences  that  may  be  present  in  the  sample. 

Several  devices  were  eliminated  from  consideration  for  various  reasons,  such  as  lack  of 
availability.  The  evaluation  panel  noted  that  biological  analysis  in  the  field  is  a  rapidly  growing 
area,  and  that  what  is  not  available  today  may  well  be  the  optimal  device  in  the  near  future. 

The  following  devices  were  evaluated: 
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Detectors 

•  BIOHAZ 

•  FACSCount 

•  LUMINEX 100 

Identifiers.  Immunoassav-based 

•  ANALYTE  2000 

•  BioDetector  (BD) 

•  BIOHAZ  with  New  Horizon  Tickets 

•  FACSCount 

•  Handheld  Assays 

•  LUMINEX  100 

•  ORIGEN  Analyzer 

•  Tetracore  Tickets 

Identifiers.  Nucleic  acid-based 

•  CEPHEID  SMART  CYCLER 

•  RAPID™  SYSTEM 

The  following  devices  were  not  evaluated  due  to  lack  of  availability  at  this  time: 

•  4WARN2 

•  ANDCARE  RAPID  GENE  DETECTION 

•  Handheld  Advanced  Nucleic  Acid  Analyzer  (HANAA) 

•  Medtox  Test  tickets  for  non-proteinaceous  toxins. 

These  devices  were  also  not  evaluated,  for  the  reasons  listed: 

•  4WARN  -  considered  to  be  an  aerosol  detector  and  not  applicable  to  the  mission. 

•  BIOCAPTURE  BT-550  -  Aerosol  sampler  w/test  tickets  -  refer  to  Handheld  Assays 
results. 

•  Guardian  BTA  Test  Str^  Reader  System  -  it  is  a  reader  that  is  associated  with 
Tetracore  tickets  and  is  not  a  stand-alone  technology. 

•  RAPTOR  -  similar  to  ANALYTE  2000 


4.  EVALUATION  MODEL 

The  foundation  of  the  evaluation  model  is  the  evaluation  criteria.  These  criteria 
encompass  the  fectors  that  most  significantly  impact  the  mission,  and  serve  to  differentiate  the 
conqjeting  candidate  technologies.  The  criteria  were  derived  from  ERT  requirements.  The 
model  was  developed  over  a  period  of  time,  on  an  iterative  basis,  as  study  team  members 
discussed  technology  capabilities  and  ERT’s  needs.  Having  all  team  members  participate  in 
criteria  development  ensured  that  customer  needs  were  expressed  in  realistic  terms  given  the 
capabilities  and  limitations  of  available  technology. 
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The  criteria  were  structured  as  a  hierarchy.  There  were  three  criteria  categories: 
EfiFectiveness,  Operational  Suitability,  and  Logistics.  Each  category  was  further  broken  down 
into  several  evaluation  criteria.  The  Sensitivity  criterion  was  further  split  into  three  sub-criteria. 
The  criteria  were  somewhat  different  depending  on  whether  the  evaluated  device  was  a  detector 
or  identifier,  which  resulted  in  two  separate  evaluation  models,  as  shown  below. 

Detector  Model 


Identifier  Model 


Best  Bio-Oet 
Identifiers 


EFFECTIVENESS 

OPER*N 

i  SUITABILITY  : 

LOGISTICS 

1 

, _ ":i _ j 

1  “ 

- 1 - 1 

1 

1 

_ 1 _ 

Material 
;  Identified 

Reagents  i  Sensitivity 

Avail?  1 

j  Reliability  | 

1  i 

i  Ease  Time  to 

of  Use  :  1  Answer 

Analysis 

Rate 

PortabHity  ;  Consumables 

-.Bacteria 

-Vims 

—Toxins 


For  each  criterion,  definitions  and  performance  scales  were  developed.  The  scales  served 
as  the  rating  scheme,  and  represented  the  different  levels  of  performance  that  could  be  expected 
for  each  criterion.  The  levels  were  assigned  values  ranging  from  0  to  100.  Two  types  of  scales 
were  developed.  Most  criteria  used  a  scale  with  defined  discrete  levels  (shown  below  for 
Portability)  in  which  devices  were  assessed  as  meeting  one  of  several  levels.  Some  criteria  used 
a  continuous  scale  (shown  below  for  Reliability)  in  v^ch  the  score  could  be  any  value  from  0  to 
100. 


•  Portability: 

100  Hand-held 

80  Man  portable 

50  Vehicle  required 

0  Not  portable 
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•  Reliability 

100  Less  likely  to  produce  felse  positive  or  negative;  siqipoiting  available 

50  Moderate  chance  of  producing  felse  positive  or  negative;  minimal  data 

available 

0  More  likely  to  produce  felse  positive  or  negative;  no  data  available 

The  final  step  in  model  development  was  to  weight  the  criteria.  The  study  team  weighted 
the  criteria  by  direct  assessment.  The  weighting  approach  involved  distributing  100  points  between 
the  fowest  level  criteria  (12  criteria  for  the  detector  model,  1 1  for  the  identifier  model). 


The  final  evaluation  model,  including  criteria  definitions,  scales,  and  weights,  is  presented  in 
^)pendix  B.  Criteria  that  were  not  included  in  the  evaluation  are  also  discussed. 

Note  that  acquisition  cost  for  the  devices  is  evaluated  separately,  as  discussed  at  the  end  of 
Sections. 


5.  EVALUATION  RESULTS 

The  evaluation  panel  assessed  the  technologies  by  scoring  each  candidate  device  against 
each  criterion,  using  the  performance  scales.  The  LDW  software  was  used  to  culture  the  consensus 
scores,  muk^ly  the  score  on  each  criterion  by  the  criterion  weight,  and  then  summing  over  all 
criteria  to  generate  an  overall  score.  The  overall  scores  were  used  to  generate  rankings,  which  was 
a  starting  point  for  the  analysis  of  the  results.  LDW  generates  a  number  of  gnqihs  and  charts  to  he^ 
analyze  the  results  and  perform  sensitivity  analysis  to  determine  vriiich  fectors  have  the  most  intact 
on  the  results.  Each  candidate  device  was  analyzed  to  assess  individual  strengths  and  weaknesses. 

This  section  summarizes  the  results.  The  results  are  presented  for  the  detector  devices  first, 
then  the  identification  devices. 

5.1  DETECTORS 

The  chart  below  shows  foe  qireadsheet  of  scores  for  the  three  detector  devices.  Note  there  is 

no  discrimination  among  the  devices  for  Process  Rate,  Portability,  and  Consumables. 


Criterion  (wt) 

Detector 

Classes 

Datsctsd 

(-20) 

Reagents 

Avail? 

(.05) 

Bacteria 

Sens 

(06) 

virus 

Sene 

(.05) 

Toxins 

(.05) 

Viability 

Deter 

(.20) 

Reliability 

(.15) 

Ease  of 
Use 

(15) 

Time  to 
first 
answer 

(.05) 

Process 

Rate 

(.02) 

Portability 

(.02) 

Consum 

-eWee 

(.01) 

BIOHAZ 

Four 

100 

100.000 

lO.OOOK 

100 

100 

90 

60 

20  or  more 

Man  port 

Low  cost 

FACSCount 

Three 

65 

lO.OOOK 

none 

75 

70 

100 

100 

2D  or  more 

Man  port 

Low  cost 

LUMINEX  100 

Three 

65 

1,000 

None 

none 

75 

70 

100 

100 

20  or  more 

Man  port 

Low  cost 

The  resulting  overall  scores  and  rankings  for  the  detector  devices  are  shown  in  the  chart 

below. 
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Detector 

BIOHAZ 
FACSCount 
LUMINEX  100 


Classes  Detected 

Reliability 

Time  to  first  answer 


til  Portability 


GS)  Viability  Deter 
m  Bacteria  Sens 
£3  Toxins  Sens 
!IB1  Process  Rate 


m  Ease  of  Use 
HI  Virus  Sens 
S  Reagents  Avail? 
m  Consumables 


This  chart  shows  how  each  device  scored,  both  overall  and  relative  to  each  criterion.  The 
vridth  of  the  sub-bars  indicates  the  weight  of  the  criterion.  For  example,  since  the  EPA’s  focus  is 
on  decontamination.  Classes  Detected  and  Viability  Determination  were  the  highest  weighted 
criteria,  and  contributed  significantly  to  each  device’s  overall  score.  However,  Biohaz  scored 
highest  on  these  criteria  and  thus  received  a  greater  contribution  to  overall  score  than  did  the 
other  devices. 

The  results  show  that  the  Biohaz  has  the  best  overall  score,  and  has  the  best  performance 
for  all  but  three  criteria.  Sensitivity  analysis  was  performed  to  see  how  the  results  would  be 
affected  by  varying  the  aiteria  weights.  This  analysis  showed  dmt  die  overall  rankings  would 
not  be  affected  by  any  reasonable  weight  changes.  Charts  that  depict  this  gr^hically  are 
contained  in  Appendix  C. 

No  detector  device  did  particularly  well  for  Vans  Sensitivity  and  Toxin  Sensitivity.  This 
is  based  on  the  sensitivity  requirements  that  were  developed  by  the  US  military  for  the 
battlefield,  and  may  not  be  ^jpropriate  for  EPA  qsplications. 

IDENTIFIERS 

TTie  chart  below  shows  the  scores  for  the  klentifier  devices. 
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Note  that  Reliability  and  Analysis  Rate  were  not  significant  discriminators.  For 

Reliability,  all  devices  scored  80  or  above.  For  Analysis  Rate,  all  but  one  device  (BioDetector) 
can  process  20  or  more  samples  in  an  hour. 

The  resulting  overall  scores  and  rankings  for  the  identifier  devices  are  shown  in  the  chart 
below.  Immunoassay-based  devices  are  denoted  by  “lA”  at  the  end  of  their  name,  and  nucleic 
acid-based  devices  are  denoted  by  “NA”. 


Identifier  Score 


Handheld  Assays  Id-IA  80 

Tetracore  tickets  Id-IA  68 

BioDetector  (BD)  Id-IA  66 

ORIGEN  Analyzer  Id-IA  65 

FACSCount  Id-IA  64 

BIOHAZ  w/  NHD  Id-IA  63 

RAPID  SYSTEM  Id-NA  57 

LUMINEX  100  Id-IA  52 

CEPHEID  SMART  CYCLER  Id-NA  47 
ANALYTE  2000  Id-IA  46 


li®  Portabili^  ■  Reliability 


•  Virus  Sens  E  Time  to  first  answer 
m  Consumables  Dll  Analysis  Rate 


Ease  of  Use 
■  Bacteria  Sens 
^  Toxins 


M^erial  Identified  (Mater  ID’d)  and  Reagent  Availability  (Reagents  Avail?)  had  the 
most  significant  impact  on  the  results  due  to  their  weight  (combined  40%  of  the  model)  and  the 
range  of  scores  amongst  the  identifier  devices. 

Handheld  Assays  had  the  highest  overall  score  because  of  its  relatively  high  score  for 
M^e^  Identified  and  Reagent  Availability,  and  high  scores  across  most  of  the  remaining 
criteria.  Its  shortcomings  included  Bacteria,  Virus,  and  Toxin  Sensitivity,  but  those  factors  were 
relatively  fow-weighted. 

Tetracore  tickets  are  comparable  to  Handheld  Assays  with  the  exception  of  having  tower 
scores  for  ferial  Identifi^  (ID’d)  and  Reagent  Availability.  This  is  because  Tetracore  has  four 
tickets  available  for  biological  agents  while  the  Handheld  Assays  have  seven.  However,  the 
performance  of  both  systems  should  be  similar. 

The  BioDetector  scored  highest  of  all  identifiers  for  the  two  most  important  criteria 
(Material  Identified  and  Reagent  Availability),  but  its  overall  score  was  reduced  by  relatively 
poor  performance  in  other  areas,  such  as  the  three  Sensitivity  criteria.  Portability,  Consumables, 
and  Analysis  Rate.  It  was  also  fek  that  this  particular  technology  may  not  be  suitable  for  this 
plication  since  it  was  designed  for  aerosols  and  may  clog  when  dirtier  samples  are  introduced. 
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Origen  and  FACSCount  had  &irly  high  scores  over  most  criteria,  but  lack  of  Reagent 
Availability  was  a  significant  problem  that  lowered  their  overall  scores.  Origen  does  score  the 
hipest  for  Toxin  Sensitivity. 

Of  the  two  nucleic  acid-based  devices,  the  Rapid  System  scored  highest,  primarily 
because  of  hi^er  Reagent  Availability  than  the  Cepheid.  Both  devices  had  the  lowest  scores  for 
Ease  of  Use,  Time  to  First  Answer,  and  Toxin  Sensitivity.  This  is  because  nucleic  acid  analysis 
techniques  require  extensive  sanq)le  preparation  and  may  be  prone  to  interferences,  such  as 
inhibitors  for  the  PCR  reaction  that  may  be  present  in  the  sample.  Both  scored  among  the  highest 
for  Bacteria  Sensitivity  and  Virus  Sensitivity.  Appendix  D  contains  a  chart  that  directly 
con[q)ares  and  contrasts  these  two  devices. 

Sensitivity  analysis  was  also  performed  for  the  identifier  devices.  The  overall  results  and 
rankings  would  not  be  affected  by  any  reasonable  criteria  weight  changes. 

.^>pendix  D  contains  an  LDW  chart  that  compares  the  highest  scoring  immunoassay- 
based  device  (Handheld  Assays)  and  the  highest  scoring  nucleic  acid-baSed  device  (R^id),  to 
^ow  bow  their  strengths  and  weaknesses  are  complementary.  Handheld  Assays  are  easier  to  use 
and  use  less  consumables,  while  the  Rapid  provides  higher  sensitivity  for  Bacteria  and  Viruses. 
This  forms  the  basis  for  certain  recommendations  discussed  in  Section  6. 

Appendix  E  contains  acquisition  cost  information  for  the  detector  and  identifier  devices, 
eTqiressed  as  a  benefit  -  cost  relationsh^,  derived  by  dividing  the  device  score  by  its  cost. 


6.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  ERT  has  multiple  requirements  in  their  two  scenarios.  No  single  device  will  be  able 
to  meet  all  requirements;  each  one  has  its  strengths  and  weaknesses.  As  a  result,  a 
con^lementary  layered  ^proach  is  recommended  that  combines  different  devices  and 
technologies.  The  basic  recommended  ^proach  is  to  screen  with  a  generic  detector,  then  use  an 
immunoassay  device  in  tandem  with  a  nucleic  acid  analj^is  technique  (PCR)  for  identificatioa 
This  provides  the  widest  possible  utility  and  capability  and  lends  credibility  and  confidence  to 
field  data.  The  specific  approach  is  described  below. 

Generic  detection  will  add  fidelity  to  the  identification  technologies  and  can  be  employed 
first  in  order  to  determine  if  a  biological  agent  is  even  present.  In  addition  it  will  altow  for  the 
detection  of  biological  materials  that  are  outside  the  library  of  identification  tools,  such  as  may 
occur  if  a  new  emerging  biological  agent  is  in  question  or  if  the  agent  that  is  used  is  not  usually 
considered  as  a  classical  BW  agent.  Most  of  the  identification  technologies  focus  on  what  are 
considered  military  threats  —  a  terrorist  has  a  wider  library  of  potential  threat  agents,  although 
their  efBcacy  may  be  questionable. 

Using  the  evaluation  model  developed  for  this  study,  the  BioHaz  Detection  system 
received  the  highest  score  in  the  generic  detector  category.  The  evaluation  panel  also  felt  that 
the  BioHaz’s  four  sampling  kits  (SWIPE  1, 2,  3,  and  4),  designed  for  taking  saitples  from 


IS 


sur&ces,  soil,  liquids,  and  air,  are  an  in:q)ortant  component  of  that  system.  This  system  can  play 
several  roles  in  the  EPA  mission.  It  can  be  used  to  screen  for  the  presence  of  these  materials  in 
suq)ect  san^les  and  can  also  be  used  to  provide  a  means  to  quality  control  decontamination 
operations,  which  is  of  chief  concern  to  the  EPA. 


Assuming  that  the  generic  screen  technique  detects  a  signature  that  may  indicate  the 
presence  of  a  biological  agent,  an  immunoassay  device  could  then  be  enq)loyed  for  presunq)tive 
identificatioa  This  is  a  mature  technology  with  an  extensive  commercial  base  thaf  is  reflected  in 
die  &ct  that  the  top  rated  choices  from  the  technology  down-select  utilize  antibody-based  assays. 
Immunoassay  systems  are  ine?q)ensive,  easy  to  use,  foirly  r^id,  often  do  not  require  extensive 
sample  preparation,  and  are  tiie  best  choice  for  the  detection  of  protein  toxins  such  as  ricin  and 
botulinum  toxin.  System  sensitivities  vary  depending  on  the  format  and  may  not  qiproach  the 
sensitivity  provided  by  ^aratus  that  target  and  amplify  the  nucleic  acid  component  of  an 
organism. 


Using  the  evaluation  model  developed  for  this  study,  the  Handheld  Assays  received  the 
highest  score  of  all  the  immunoassay-based  devices  evaluated.  This  system  utilizes  hand-held 
immuno-chromatographic  assays  that  are  str^based  antibody  tests  similar  in  design  to  home 
pregnancy  tests.  These  assays  are  available  for  an  expanding  library  of  agents  that  could  be  used 
as  biotogical  we^ns.  One  potential  shortcoming  that  needs  to  be  addressed  is  that  these 
reagents  are  only  available  through  the  military  and  may  have  limited  availability  to  non-DOD 
users.  Howwer,  there  are  kits  that  are  commercially  available  (Tetracore  and  New  Horizons) 
that  have  similar  performance  c^abilities,  though  with  a  smaller  library  of  target  agents. 

The  third  and  final  system  recommended  is  nucleic  acid-based.  Systems  that  iiri1iyj»  PCR 
and  fluorescence-based  detection  of  nucleic  acid  hybridization  can  detect  and  identify  bacteria 
and  viruses  that  carry  DNA  within  them,  assuming  that  no  inhibitors  were  introduced  into  the 
analysis.  Th^  systems  may  be  able  to  detect  residual  DNA  that  binds  to  toxin  preparations, 
such  as  botulinum  toxin,  if  the  toxin  san:q)le  has  not  been  subjected  to  conditions  that  may  cans? 
exte^ive  degradation  or  removal  of  the  contaminant  DNA.  This  technology  is  exquisitely 
specific  and  sensitive  and  is  usually  not  prone  to  felse  positives.  Detection  strategies  are 
^igned  so  that  mult^le  locations  within  the  genomes  of  a  bio-threat  agent  are  targeted.  This 
iricreases  the  confidence  of  the  identification  and  may  even  be  able  to  provide  information  on  the 
virulence  of  ^  biological  agent,  if  the  q>propriate  genes  are  present  and  probed  for.  Thus,  pcR 
can  differentiate  a  vaccine  strain  from  a  we^>ons  strain  of  Bacillus  anthracis  and  confirm 

th^  antibody-based  tests  were  accurate  in  their  results.  PCR  technology  is  becoming  more 
widely  ^li^  in  field  plications  as  n^gedized  units  have  become  commercially  available. 
However,  this  technology  does  require  significant  sample  preparation  and  can  be  inhibited  by 
some  environmental  contaminants. 


Using  the  evaluation  model  developed  for  this  study,  the  RAPID™  received  the  highest 
score  in  the  category  of  PCR-based  devices.  A  number  of  freeze-dried  PCR  assays  which  target 
the  DNA  or  RNA  of  biological  weapons  are  already  available  and  that  number  is  expected  to 
grow  significantly  during  the  next  two  years.  The  system  can  operate  from  a  battery  and  linV 
tq)  with  the  internet  giving  the  RAPID  an  autonomous  web  based  ability  to  transmit  to 

remote  sites  for  sharing  with  subject  matter  experts  not  collocated  with  the  response  teams.  The 
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RAPID  ™  requires  a  sample  prq)aration  c^ability  that  has  been  developed  by  ECBC  for  other 
customers  for  use  in  the  field.  This  system  is  also  used  by  the  Biological  Analysis  Teams  (BAT) 
that  was  established  by  the  US  Air  Force. 

The  generic  detector,  immunoassay,  and  PCR  devices  should  be  combined  together  to 
form  the  core  of  the  ERT  biological  detection  c^ability.  In  this  approach,  a  great  many  foctors 
have  been  considered,  including  the  con^)lementary  information  of  each  of  the  technologies  with 
one  another,  the  wide  array  of  tasks  that  may  come  into  play,  and  the  time  and  technical 
considerations  that  a  field  team  is  likely  to  encoimter  when  dealing  with  a  series  of  unknown 
samples.  The  final  decision  for  technologies  can  be  supported  by  a  decision  based  methodology 
manual  ^cific  for  the  ERT  concept  of  operations.  Training  in  these  systems  could  augment 
that  ERT  based  methodology  and  would  provide  the  EPA*s  Emergency  Response  Teams  with  a 
biological  cspability  of  sufficient  scope  to  meet  their  objectives. 
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APPENDIX  A 


STUDY  TEAM  MEMBERS 


Customer  lEPA-ERT)  Representatives 

Harry  Compton 
Rod  Turpin 
Phil  Caii^agna 
David  Mickunas 
Geoffrey  Betsinger 

Technical  Subject  Matter  Experts  (Evaluation  Panel) 

Jerry  Bottiger,  Ph.D. 

Peter  Emanuel,  Ph.D. 

Michael  Goode 
Gil  Olsen 
Kate  Ong 
Peter  Stopa,  Ph.D. 

Peter  Snyder,  Ph.D. 

Project  Management/Customer  Interface 

Dennis  Boh 
Randy  Laye 
Uday  Mehta 

Decision  Analysts 


John  Walther 
Valerie  Outlaw  Lee 
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APPENDIX  B 


EVALUATION  MODEL 


The  criteria  are  grotq)ed  into  three  major  categories:  Effectiveness,  Operational 
Suitability,  and  Logistics.  Each  category  has  two  to  four  sub-criteria.  Some  evaluation  criteria 
are  specific  to  either  detection  or  identification,  resulting  in  two  separate  models.  Weights  are 
shown  in  parentheses;  a  “D”  indicates  the  weight  qiplies  to  the  Detector  model,  an  “F*  indicates 
the  weight  ^lies  to  the  Identifier  model,  while  no  designation  means  the  weight  is  the  same  for 
either  model. 


EFFECTIVENESS  —  How  well  the  detector  performs  (technical  characteristics). 

1 .  Number  of  Classes  Detected  or  Materials  Identified 

A.  Number  of  Classes  Detected  (.20,  D):  Number  of  different  biological  classes  the  devices 
can  detect  (the  classes  listed  below  are  listed  in  priority  order).  This  criterion  is  used  only  for  the 
devices  classified  as  detection  devices. 

1 00  Sporulated  Bacteria,  Vegetative  Bacteria,  Toxins,  and  Viruses 

65  Three  classes 

35  Two  classes 

0  One  class 

B.  Number  of  Biological  Materials  Identified/Assays  Developed  (.20, 1):  The  number  of 
different  biological  agents  that  can  be  identified  by  the  device,  using  the  following  generally 
accepted  bio  agents  (traditional  agents,  commonly  accepted  wesqponized  DoD  targets)  as  a 
baseline: 

NOTE  -  the  specific  agents  used  as  the  reference  for  the  evaluation  have  been  removed  from  the 
report  to  avoid  potential  information  classification  issues.  The  specific  agents  that  can  be 
identified  by  specific  devices  can  be  obtained  from  the  study  sponsor. 

This  criterion  is  used  only  for  the  devices  classified  as  identification  devices. 


100 

All  eight 

87 

Seven 

75 

Six 

62 

Five 

50 

Four 

37 

Three 

25 

Two 

12 

One 

0 

None 
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1.  Reagent  AvaQability 


“AvMlable”  means  through  commercial  or  government  (JPO)  sources,  in  some 
substantial  quantities,  in  the  form  of  a  con:5)leted  end  item.  For  detector  devices,  the  scale  is 
the  same  as  1 A  (weight  of  .05),  for  identification  devices,  the  scale  is  the  same  as  in  IB 
(weight  of  .20). 

2.  Sensitivity:  The  minimum  amount  of  BW  agent  that  must  be  present  for  the  device  to  make 
a  positive  detection  and/or  identificatioa 

3a.  Bacteria  (.05) 

100  1,000  CFU/mL 

75  10,000  CFU/mL 

25  100,000  CFU/mL 

10  1,000,000  CFU/mL 

0  No  detection 

3b.  Toxins  (.05) 

100  0.1  ng/mL 

75  1.0  ng/mL 

25  10  ng/mL 

10  100  ng/mL 

0  No  detection 

3c.  Viruses  (.05) 

With  the  exertion  of  BioDetector  data  is  based  on  A£S2  virus  simulant 

100  10,000  PFU/mL 

85  100,000  PFU/mL 

70  1,000,000  PFU/mL 

40  10,000,000  PFU/mL 

10  100,000,000  PFU/mL 

0  No  detection 


^^bility  (.20,  D)  —  ability  to  determine  presence  of  “dead  or  live”  organisms. 
This  criterion  is  used  only  for  the  devices  classified  as  detection  devices. 

100  detects  viability  of  4  classes 

75  detects  viability  of  3  classes 

50  detects  viability  of  2  classes 

25  detects  viability  of  1  class 

0  detects  no  viability 
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OPERATIONAL  SUIT  ABILITY  -  Ability  to  operate  the  detector  in  the  field,  including 
impact  on  operations. 

5.  Reliability  (.15  D,  .10 1)  -  tendency  for  the  identification  devices  to  provide  felse  positives  or 
false  negatives. 

Scores  based  primarily  on  expert  judgement,  as  little  actual  data  is  available. 

100  Less  likely  to  produce  felse  positive  or  negative;  supporting  data  available 
50  Moderate  chance  of  producing  felse  positive  or  negative;  minimal  data  available 

0  More  likely  to  produce  felse  positive  or  negative;  no  data  available 

6.  Ease  of  Use  (.15  D,  .10 1):  The  number  and  conq>lexity  of  steps  required  to  collect  and 
prepare  the  sample  and  perform  the  assay.  Intended  user  assumed  to  have  a  technical 
bachelor’s  degree  with  training. 

1 00  Low  opportunity  for  error  (e.g.,  minimal  number  of  steps) 

60  Medium  opportunity  for  error 

0  High  opportunity  for  error  (e.g.,  significant  number  of  complex,  hand- 

manipulated  operations) 

7.  Time  to  first  answer  (.05):  The  time  required  to  uiqiack  the  detector  and  have  it  operational 
in  the  field,  prepare  and  analyze  one  soil  sample  (soil  is  the  most  extreme  situation)  against 
four  agents  to  yield  a  response.  Includes  instrument  equilibration  and  reconstituting  reagents. 


100  ^  30  minutes 

80  >  30  minutes  to  1  hour 

60  >1  hour  to  <  2  hours 

40  >2  hours  to  ^  3  hours 

20  >3  hours  to  <  4  hours 

0  >4  hours 


8.  Rate  (Process  Rate  for  detectors  (.02),  Instrument  Analysis  Rate  for  identifiers  (.05)): 
The  number  of  soil  samples  (worst  case)  that  can  be  assayed  per  hour,  not  including  sample 
prep. 

100  20  or  more  samples  per  hour 

50  10-20  sample  per  hour 

25  5-9  samples  per  hour 

0  less  than  5  san^les  per  hour 
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LOGISTICS  -  Support  required  to  transport,  set-up,  and  operate  the  device  in  the  field. 

9.  Portability  (.02  D,  .10 1):  Ease  of  transporting  the  detector  to  the  field  and  moving  and 
operating  the  unit.  Includes  device  and  operational  supplies  (i.e.,  the  detector,  sandier, 
required  kits,  other  support  equipment).  Note  -  All  devices  will  require  some  protective 
operational  environment  (50-80  deg  F) 


100 

Hand-held 

80 

Man  portable 

50 

Vehicle  required 

0 

Not  portable 

10.  Consumables  (.01  D,  .05 1)  -  The  cost  of  reagents  and  support  supplies  (sampling  kits, 
plastic  tubes,  etc.)  to  perform  one  assay  for  one  agent  (including  sample  prep  and  clean-up, 
not  including  sample  acquisition).  The  devices  will  require  different  number  of  assays. 

100  Low  $20 

50  Medium 

0  High  >$50 


1 1.  Overall  Instrument  Cost:  cost  of  instrument  and  sampler,  if  necessary. 
Tltis  is  assessed  separately  (not  part  of  the  evaluation  model);  see  Appendix  E. 
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The  following  criteria  were  eliminated  from  the  evaluation  model;  the  rationale  is 
italicized: 

Specificity:  Bacteria  vs.  Aerosol:  The  ability  of  the  detector  to  identify  the  specific  BW  agent 
versus  detecting  an  aerosol  in  the  sanq)le  only. 

100  Specific  agent  detection 
60  Agent  classification  (bio/virus/toxin) 

0  Aerosol  detection  only 

Specificity  is  d^ned  towards  aerosols  as  most  likely  seen  on  the  battlefield,  and  is  covered  by 
other  factors  in  the  model.  Aerosol  detection  is  where  most  of  these  devices  have  been  used,  to 
date. 

Technical  Field  Support:  Availability  of  technical  siq>port  (engineering  and  scientific  stqjport) 
viien  the  detector  is  in  tiie  field. 

100  Immediate  on-site  assistance  available 

80  Remote  diagnostic  assistance  via  phone 

60  Delayed  on-site  assistance 

40  Internet  web  site 

0  No  assistance  likely 

Off-site  assistance  may  be  available  through  the  Web;  not  vendor  specific.  There  is  not  enough 
information  available  about  the  vendors  to  determine  the  type  of  support  available,  although  the 
subject  matter  apert  panel  felt  that  all  devices  will  have  at  least  minimal  support  available 
^}erfuq}S  phone). 

Durability:  ability  of  the  instrument  to  consistently  perform  its  intended  frmctions  (mobile  lab  in 
a  field  setting). 

1 00  can  be  frilly  operated  in  a  non-protected  environment 

80  can  be  fully  operated  out  the  back  of  a  small  van 

0  requires  lab  type  setting 

The  subject  matter  expert  panel felt  all  the  devices  were  comparable  in  terms  of  Ruggedness  or 
Durability  (pachiging  design  should  resolve  ar^  concerns),  so  this  criterion  would  not  help  to 
discriminate  between  the  alternative  devices. 

Power:  The  type  of  power  required  for  the  detector. 

100  No  power  required 

90  Battery  powered 

60  1 10  VAC  power  source  required 

0  Power  from  support  vehicle  220  volt;  special  power  requirements 

The  subject  matter  expert  panel  felt  the  devices  were  comparable  in  that  most  will  operate  on 
vehicle  or  separate  battery  power. 

Ability  to  process  various  sample  types  (air,  liquid,  soiQ. 

Addressed  sufficiently  in  Time  to  First  Answer  and  Reliability. 
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APPENDIX  C 


SENSITIVITY  CHARTS  FOR  DETECTORS 


The  three  charts  below  show  the  sensitivity  analysis  for  the  criteria  v^^ere  Biohaz  did  not 
score  highest  (Ease  of  Use,  Bacteria  Sensitivity,  and  Time  to  First  Answer).  For  each  chart,  the 
relative  overall  ranking,  given  the  current  weighting,  is  shown  where  the  vertical  line 
(representing  the  current  weight)  intersects  each  of  the  three  device  lines. 

Ease  of  Use  was  die  most  sensitive,  but  the  weight  would  have  to  be  increased  from  15% 
to  25%  (seen  by  shifting  the  vertical  weight  line  to  the  right)  before  the  overall  results  change. 
Bacteria  Sensitivity  would  have  to  more  than  double  its  current  weight  of  5%  before  FACSCount 
would  be  preferred  overall.  Time  to  First  Answer  would  require  an  unrealistic  increase  to 
change  the  overall  results. 


-  BIOHAZ  Det 

FACSCoimtDet 
---  LUMINEXlOODet 


27 


-  BIOHAZDet 

FACSCount  Det 
-  LUMINEXlOODet 


Percent  of  Weight  on  Bacteria  Sens  Measure 
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APPENDKD 


SENSITIVITY  CHARTS  FOR  IDENTIFIERS 


The  chart  below  directly  con^ares  the  R^id  and  Cepheid  devices.  The  only  difference 
in  the  scoring  was  Reagent  Availability  and  Reliability. 


Overall  Score  for 

RAPID  SYSTEM  Id-NA 

57 

CEPHEID  SMART  CYCLER  Id-NA 

47 

Difference 

11 

Total  Diffetence 
Reagents  Avail? 
Reliability 


CEPHEID  SMART  CYCLER  Id-NA 


RAPID  SYSTEM  Id-NA 
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The  two  charts  below  con^are  the  highest  scoring  immunoassay-based  device  (Handheld 
Assa}^)  and  the  highest  scoring  nucleic  acid-based  device  (R^)id),  to  show  how  their  strengths 
and  weaknesses  arc  con5)lementary.  Handheld  Assays  are  easier  to  use  and  use  less 
consumables,  while  the  R^id  provides  higher  sensitivity  for  Bacteria  and  Viruses. 


Measure  Scores  for  Handheld  Assays  Id-IA  for  BEST  BIO  DET  Identifier  Goal 


■  Reagents  Avail? 
HI  Portability 

■  Consumables 
□  Virus  Sens 


[H]  Mater  nyd 
■  Ease  of  Use 
□  Toxins  Sens 
ID  Bacteria  Sens 


01  Reliability 
■  Analysis  Rate 
S  Time  to  first  answer 


Measure  Scores  for  RAPID  SYSTEM  Id-NA  for  BEST  BIO  DET  Identifier  Goal 

100  I  mm 


■  Reagents  Avail?  □  Mater  ID'd 
Hi  Portability  I  Ease  of  Use 

m  Consumables  H  Toxins  Sens 

D  Virus  Sens  m  Bacteria  Sem 


E3  Reliability 
■  Analysis  Rate 
§  Time  to  first  answer 


APPENDIX  D 


30 


APPENDIX  E 


BENEFIT  -  COST  ASSESSMENT 


This  table  provides  a  comparison  of  each  device’s  “benefit”  (the  score  from  the  assessment) 
relative  to  the  acquisition  cost.  Note  the  cost  does  not  include  consumables. 

Within  each  category,  the  devices  are  listed  in  descending  order  of  Benefit/Cost  ratio. 


Device 

Score 

Cost  ($K) 

Benefit/Cost 

Detectors  ^  - 

BIOHAZ 

80 

20 

4.0 

LUMINEX  100 

71 

37 

1.9 

FACSCount 

73 

59 

1.2 

Ideritffiers,  Immunodssdy-based  * 

Alexeter  BTA  (Tetracore  tickets) 

68 

4.5 

15.1 

ANALYTE  2000 

46 

14 

3.3 

BIOHAZ  with  New  Horizon  Tickets 

63 

20 

3.2 

ORJGEN  Analyzer 

65 

46 

1.4 

LUMINEX  100 

52 

37 

1.4 

FACSCount 

64 

59 

1.1 

BioDetector  (BD) 

66 

125 

0.5 

itd&ittfiers.  Nucleic  acid-based  • 

CEPHEID  SMART  CYCLER 

46 

35 

1.3 

RAPID  SYSTEM 

57 

55 

1.0 

*  The  Handheld  Assays,  which  scored  80,  is  not  listed  because  it  is  a  govemraent-fumished 
product,  and  the  cost  must  be  determined  by  the  responsible  government  agency. 


